Digital camera incorporating a sharpness predictor

ABSTRACT

A method for determining a sharpness predictor for an input digital image includes determining one or more image metrics by analyzing the input digital image; and determining the sharpness predictor from the one or more image metrics.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.10/938,770 filed Sep. 10, 2004 now U.S. Pat. No. 7,764,844, which isincorporated herein by reference in its entirety.

FIELD OF THE INVENTION

This method pertains to the field of digital imaging, and moreparticularly to the field of determining a sharpness predictor for aninput digital image.

BACKGROUND OF THE INVENTION

When images are captured by a camera, there are a number of factors thatcan adversely affect the quality of the resulting image. One importantfactor is the sharpness of the image. There are several attributes ofthe imaging system that affect the sharpness of the image, such as thequality of the lens and the MTF of the image capture medium. But beyondthese fundamental characteristics, there are other image-dependentfactors that can seriously degrade the sharpness, and therefore theimage quality, of the image. These factors include overall lens defocus(e.g., due to attempting to capture an image of an object that is tooclose to the camera), defocus of the main subject (e.g., due to thecamera auto-focusing on the wrong object in the scene), insufficientdepth-of-field to capture the range of important object distances,fogging of the lens element due to condensation of water vapor, andsmearing due to motion of the camera and/or objects in the scene.

In conventional photography using photographic film, it is usually notpossible to determine whether the captured image has acceptablesharpness until the film is processed and printed. With the advent ofdigital cameras, it is possible to preview the image at the time ofcapture. In theory, this allows the photographer to assess the qualityof the image, and, if necessary, capture another image to correct imagequality problems. However, the quality of the preview displays used onmost digital cameras is insufficient to adequately evaluate the imagequality in many cases. As a result, the photographer may not realizethat the quality of an image is poor until after the image has beenprinted or previewed at a later time on a high-resolution display. As aresult, the photographer may miss any opportunity to capture an improvedimage, or may be dissatisfied that time/money was wasted in making aprint of a poor quality image. Therefore, there is a need for a means toautomatically assess the sharpness of a digital image at the time thatthe image is captured.

There are examples of prior art in this field. Some cameras offer awarning signal if camera shake is likely to occur due to an excessivelylong exposure time. Tomita discloses (U.S. Patent ApplicationPublication 2004/0018011 A1) a vibration correcting optical devicecomprised of a vibration detection unit that detects a vibration of thevibration correcting optical device and outputs a vibration detectionsignal corresponding to the vibration. Wheeler, et al. disclose (U.S.Patent Application Publication 2003/0095197 A1) a method of using imagemetadata to predict camera shake. A significant problem with thesearrangements is that they do not directly measure the level of blur inthe captured image itself, but rather attempt to predict whether theimage might be blurred based on other pieces of information. As aresult, these methods are only effective at identifying certain sourcesof image blur (e.g., blur due to camera shake).

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a sharpnesspredictor that can be used to detect digital images having poorsharpness.

This object is achieved by a method for determining a sharpnesspredictor for an input digital image comprising:

a) determining one or more image metrics by analyzing the input digitalimage; and

b) determining the sharpness predictor from the one or more imagemetrics.

It is a feature of this invention that this sharpness predictor can beused in applications such as alerting a user that a digital image mayhave poor sharpness in order to provide an opportunity to capture animproved digital image, or that the digital image may not be appropriatefor producing high-quality prints.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart showing the method of the present invention; and

FIG. 2 shows four different discrete consine transforms.

DETAILED DESCRIPTION OF THE INVENTION

Turning to FIG. 1, the present invention will be described withreference to a preferred embodiment. A method is described fordetermining a sharpness predictor for an input digital image 10. First,the input digital image 10 is analyzed using one or more image analysissteps 20 to determine one or more corresponding image metrics 30,wherein at least one of the image metrics 30 is related to the spatialfrequency content of the input digital image 10. Next, a determinesharpness predictor step 40 is used to determine a sharpness predictor50 from the one or more image metrics 30, wherein the sharpnesspredictor 50 represents an estimate of the sharpness of the inputdigital image 10. Each of these steps will now be discussed in moredetail.

In a preferred embodiment of the present invention, the input digitalimage 10 is an image captured using a digital camera. The invention canbe applied to a digital image at any point in the imaging chain of thedigital camera. For example, the input digital image 10 can be a rawdigital image corresponding to the initial digital image captured by thesensor in the digital camera. Alternatively, a processed digital cameraimage can be used for the input digital image 10, where the processeddigital camera image is formed by applying various processing steps tothe raw digital image. The processed digital camera image can correspondto the output digital image formed by the digital camera (e.g., an imageprepared for display on a conventional CRT in a color space such as thewell-known sRGB), or it can correspond to an intermediate processingstate. It will be obvious to one skilled in the art that other forms ofinput digital images 10 could also be used in accordance with thepresent invention, including digital images formed by scanningphotographic prints or negatives.

The present invention can be applied to color input digital images orblack-and-white input digital images. For the case of a color inputdigital image, the present invention is preferably applied to aluminance signal extracted from the color input digital image since mostof the sharpness information in an image is contained in the luminanceportion of the image (although it can also be applied to the full-colorinput digital image). A luminance signal is typically determined fromthe color input digital image by computing a weighted sum of theindividual color channelsY=K _(r) R+K _(g) G+K _(b) B  (1)where R, G and B are the red, green and blue color channel values for aparticular pixel of the color input digital image, and K_(r), K_(g) andK_(b) are weighting coefficients. There are many different sets ofweighting coefficients that are commonly used in the art for determininga luminance signal depending on the input color space and the particularapplication. In a preferred embodiment of the present invention, theinput color space is a video RGB color space (such as the well-knownsRGB), and a corresponding set of weighting coefficients areK_(r)=0.299, K_(g)=0.587 and K_(b)=0.114

There are many different forms of image analysis steps 20 that can beused in accordance with the present invention to determine the one ormore image metrics 30. Since the sharpness of the image is fundamentallyrelated to the spatial frequency content of the input digital image 10,it will generally be desirable for the image metrics 30 to infer imagesharpness by analyzing the input digital image 10 to estimate thespatial frequency content of the input digital image 10. There are manydifferent ways to estimate the spatial frequency content of the inputdigital image 10. In a preferred embodiment of the present invention, adiscrete cosine transform (DCT) is performed on blocks of the inputdigital image 10 to compute blocks of discrete cosine transformcoefficients. These blocks of discrete cosine transform coefficientsprovide estimates of the spatial frequency content contained in thecorresponding block of the input digital image 10. This is convenient inmany cases since many digital cameras already use DCTs in their imagingchains as part of an image compression operation. Therefore, the cameraswill frequently have software, or dedicated hardware, available toperform these calculations. Alternatively, other spatial frequencyestimators, such as Fast Fourier Transforms (FFTs), gradient estimatoroperators, or wavelet transforms, could be used to estimate the spatialfrequency content of the input digital image 10. In general, it may beadvantageous from a computational efficiency standpoint to utilizespatial frequency estimators that are calculated for other reasons, suchas image compression operations, in the process of computing the imagemetrics 30.

The two-dimensional DCT of an n×n block of input digital image pixels,f_(i)(j,k), is given by:

$\begin{matrix}{{F_{i}\left( {u,v} \right)} = {\frac{4{C(u)}{C(v)}}{n^{2}}{\sum\limits_{j = 0}^{n - 1}{\sum\limits_{k = 0}^{n - 1}{{f_{i}\left( {j,k} \right)}{\cos\left\lbrack \frac{\left( {{2j} + 1} \right)u\;\pi}{2n} \right\rbrack}{\cos\left\lbrack \frac{\left( {{2k} + 1} \right)v\;\pi}{2n} \right\rbrack}}}}}} & (2)\end{matrix}$where i is the block number, j and k are the horizontal and verticalcoordinates of a pixel in the block of input digital image pixels,respectively, u and v are the horizontal and vertical spatial frequencyindices, respectively,

$\begin{matrix}{{C(w)} = \left\{ \begin{matrix}{{1/\sqrt{2}};} & {{{for}\mspace{14mu} w} = 0} \\{1;} & {{{{for}\mspace{14mu} w} = 1},2,\ldots\mspace{14mu},{n - 1}}\end{matrix} \right.} & (3)\end{matrix}$and F_(i)(u,v) is the resulting block of discrete cosine transformcoefficients. The values of F_(i)(u, v) represent the amplitudes of thespatial frequency content for the i^(th) image block at the frequencies(u, v).

Once the DCT coefficients have been computed for each block of the inputdigital image 10, a variety of image analysis steps 20 can be used todetermine corresponding image metrics 30. Many of these image metrics 30can be determined by computing various image statistics from the blocksof DCT coefficients. Examples of several such image statistics will nowbe described. However, it will be obvious to one skilled in the art thatmany different forms of image analysis steps 20 could be used inaccordance with the present invention.

The sharpness of the input digital image will be reflected in the amountof high spatial frequency information in the DCT coefficients. One imagestatistic, which is a measure of how much high spatial frequencyinformation is present, is to calculate the radial frequency of thecentroid of the DCT coefficients for each block of input digital imagepixels. This will provide a measure of how much high spatial frequencycontent there is in each image block. The values from each image blockcan then be combined to form a single overall DCT centroid image metric,M₁:

$\begin{matrix}{M_{1} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}\left\lbrack \frac{\sum\limits_{u = 1}^{n}{\sum\limits_{v = 1}^{n}{{{F_{i}\left( {u,v} \right)}}\left( {u^{2} + v^{2}} \right)^{1/2}}}}{\sum\limits_{u = 1}^{n}{\sum\limits_{v = 1}^{n}{{F_{i}\left( {u,v} \right)}}}} \right\rbrack}}} & (4)\end{matrix}$where N is the number of image blocks. Generally, it is desirable toexclude the DC coefficient (i.e., u=v=1) from this calculation so thatthe image metric is independent of the overall brightness of the imageblock. This can be done by setting F_(i)(1,1)=0 before computing theimage metric.

The image metric shown in Eq. (4) is sometimes prone to misidentifyimage blocks as being unsharp, when in reality they simply correspond toflat areas of the image such as skies. An improvement to this metric canbe made by only including image blocks in the calculation that containsignificant spatial frequency content. Another image statistic, which isan indicator of whether an image block contains significant spatialfrequency content, is the maximum amplitude of the DCT coefficients.This quantity can be determined and compared to some threshold T toidentify the image blocks that should be included in the computations:Max(|Fi(u,v)|)>T  (5)Any image block whose DCT coefficients are all smaller than thethreshold value, T, will not be included in the computation of imagestatistics. While the image metric in Eq. (4) of the Centroid canidentify DCT blocks having higher amounts of high frequency information,this is a relative metric. If two blocks have the same value for Eq. (4)but different overall DCT coefficient response levels, the DCT blockhaving the greater overall DCT coefficient response level is moresignificant. Therefore an image statistic that sums the total DCTresponse in the block, after zeroing the DC contribution, is desirable.The values from each image block can then be combined to form anintegrated DCT response image metric, M₂:

$\begin{matrix}{M_{2} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}\left( {\sum\limits_{u = 1}^{n}{\sum\limits_{v = 1}^{n}{{F_{i}\left( {u,v} \right)}}}} \right)}}} & (6)\end{matrix}$where N is the number of image blocks in the image.

The image metrics described in Eqs. (4) through (6) can be extended tosubregions of the DCT block as follows. The DCT block can be envisionedas including subregions corresponding to all low frequency content, allhigh frequency content and sub-regions of mixed high and low frequencycontent in either the horizontal or vertical directions. One possiblemethod of partitioning the DCT block into such subregions would be todivide the DCT block into four equal quadrants. It will be obvious toanyone skilled in the art, that other partitionings would be possibleand all such partitionings share a common interpretation.

FIG. 2, shows a DCT block quadrant partitioning, including subregionscorresponding to all low frequency content 100, all high frequencycontent 130, and a mixture of high horizontal frequency content and lowvertical frequency content, 110, or high vertical frequency content andlow horizontal frequency content, 120. Under such a partitioning schemethe radial frequency can be computed for each quadrant of the DCT block.This will provide a measure of how much high spatial frequency contentthere is in each quadrant of the image block. The values from eachquadrant of the image block are then be combined to form a singleoverall image metric appropriate for that quadrant, M_(1j):

$\begin{matrix}{M_{1j} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}\left\lbrack \frac{\sum\limits_{u_{j} = 1}^{n_{j}}{\sum\limits_{v_{j} = 1}^{n_{j}}{{{F_{ij}\left( {u,v} \right)}}\left( {u_{j}^{2} + v_{j}^{2}} \right)^{1/2}}}}{\sum\limits_{u_{j} = 1}^{n_{j}}{\sum\limits_{v_{j} = 1}^{n_{j}}{{F_{ij}\left( {u_{j},v_{j}} \right)}}}} \right\rbrack}}} & (7)\end{matrix}$where j=1, 2, 3, or 4, N is the number of image blocks in the image andn_(j) is the number of DCT coefficients along either the horizontal orvertical dimension of the j^(th) DCT block quadrant. This would provideincreased resolution in determining the difference in high spatialfrequency distribution within the DCT block, and would then replace thesingle overall DCT centroid image metric M₁ with four quadrant DCTcentroid image metrics M₁₁, M₁₂, M₁₃, and M₁₄. In similar fashion, thevalues of each quadrant of the image block can also be combined to forman integrated DCT coefficient response metric appropriate for thatquadrant, M_(2j):

$\begin{matrix}{M_{2j} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}\left( {\sum\limits_{u_{j} = 1}^{n_{j}}{\sum\limits_{v_{j} = 1}^{n_{j}}{{F_{i}\left( {u_{j},v_{j}} \right)}}}} \right)}}} & (8)\end{matrix}$where j=1, 2, 3, or 4, N is the number of image blocks in the image andn_(j) is the number of DCT coefficients along either the horizontal orvertical dimension of the j^(th) DCT block quadrant. This would provideincreased resolution in determining how the total integrated DCTcoefficient response is distributed within each DCT block, and wouldthen replace the single integrated DCT coefficient response metric M₂with four quadrant integrated DCT coefficient response image metricsM₂₁, M₂₂, M₂₃, and M₂₄.

The example image metrics that have been described above areillustrative of image metrics 30. However, it will be obvious to oneskilled in the art that many other forms of image metrics could also bedetermined that could be used in accordance with the present invention.Examples of other types of image metrics 30 that could be used wouldinclude the number of image pixels darker than a specified threshold,the number of image pixels lighter than a specified threshold, edgegradients, contrast changes, or pixel kurtosis.

Once all of the image metrics 30 have been computed, the determinesharpness predictor step 40 is used to determine the sharpness predictor50. In a preferred embodiment of the present invention, the imagemetrics 30 can be combined by computing a weighted summation:

$\begin{matrix}{P = {\sum\limits_{l = 1}^{L}{a_{l}M_{l}}}} & (9)\end{matrix}$where L is the number of image metrics, a_(l) is a weighting coefficientused to weight an l^(th) image metric M₁. It should be noted each of theimage metrics M₁ could be individual image metrics such as thosedescribed above, or could be some linear or nonlinear combination ofsuch image metrics. For example, M_(c) can be determined by multiplyingM_(a) and M_(b3). This also provides a means to include interactionterms. Similarly, higher-order terms can be included by including imagemetrics that are powers of previously defined image metrics (e.g.,M_(d)=(M_(a2))²). Similarly, new image metrics can be computed that arefunctions of the various image metrics. For example, a new image metriccould be computed that is the logarithm of another image metric. Aweighted sum of such functions of the one or more image metrics couldthen be included in the weighted summation in Eq. (9).

The weighting coefficients, a_(l), from Eq. (9) can be determined invarious ways. Many of these methods rely on the existence of a set oftest images for which an independent assessment of unsharpness alreadyexists. This assessment should be continuous in nature and be relativelyuniform across the image space. If a change in unsharpness were to occurin any image assessed at low levels of image unsharpness, the visualimpact of this change in unsharpness should be approximately the same asit would be if it were to occur in an image assessed at high levels ofunsharpness. One possible scale that possesses this property is thewell-known Just Noticeable Difference (JND) scale. Once a set of testimages have been assembled and subjectively evaluated in some manner toassign to each image a JND value reflecting the subjective evaluation ofunsharpness of this image, the same set of images can then be analyzedin accordance with the methods described above to determinecorresponding image metrics 30, and any appropriatecombinations/transformations of these image metrics. Statisticalmethods, such as linear regression, can then be used to correlate valuesof the two or more image metrics, 30, and the subjectively determinedimage sharpness values, thereby determining the least squares bestestimates of the weighting coefficients, a_(l). The linear regressionresults in weighting coefficients, a_(l), which, in conjunction with theimage metrics, M_(l), constitute a determine sharpness predictor step40. In a preferred embodiment of the present inventions, application ofthe weighting coefficients, a_(l), to the image metrics, M_(l), in themanner described in Eq. (9), generates sharpness predictor 50 for animage with units of JNDs of unsharpness.

Additionally, this continuous scale of JND values could then besegmented to generate categories of unsharpness appropriate fordetermining whether the quality of the input digital image 10 issufficient for multiple applications. A possible application might bethe rendering of the input digital image 10 as a reflection print.Determination of the unsharpness category associated with the inputdigital image 10 could result in a warning message, or a recommendedimage usage message, being sent to the user via a user interface, orother appropriate action taken. This information could also be stored inthe file header information associated with the input digital image 10,in the digital camera, to be used as a control parameter to facilitatetaking corrective actions at the time of printing the input digitalimage 10 to achieve optimal image quality.

It will be obvious to one skilled in the art that other methods can beused to combine the image metrics 30 to determine the sharpnesspredictor 50. For example, one alternative approach would be to use aso-called “neural network” model. Such models are well known in the art.Such methods could also include the use of sharpness metrics determinedfrom image capture parameters associated with the digital camera. Suchimage capture parameters could include an exposure time parameter, alens aperture parameter, a flash-fired parameter, or combinationsthereof. These image capture parameters would then be used as additionalimage metrics 30 in the manner described above to determine thesharpness predictor 50.

The steps described above could also be applied to any subset of pixelsextracted from the input digital image 10 and the image metrics couldthen be determined from this subset of the input digital image. Forexample, a subset of the input digital image extracted from the centerof the input digital image could be used to select the image regionlikely to be of greater importance to the user and the image metricscould then be determined from this subset of the input digital image.Alternatively, input digital image pixel subsets could be selected byvarious preprocessing criterion aimed at identifying regions of enhancedinterest. For example, a subset of the input digital image can beinteractively specified by a user, or the main subject of the imagecould be identified using an automatic main subject detection algorithm.This input digital image pixel subset would then become the new inputdigital image 10, and the analysis procedure would proceed as describedabove.

A computer program product can be used in the practice of this inventionand includes one or more storage medium, for example; magnetic storagemedia such as magnetic disk (such as a floppy disk) or magnetic tape;optical storage media such as optical disk, optical tape, or machinereadable bar code; solid-state electronic storage devices such as randomaccess memory (RAM), or read-only memory (ROM); or any other physicaldevice or media employed to store a computer program having instructionsfor controlling one or more computers to practice the method accordingto the present invention.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   10 input digital image-   20 analysis steps-   30 image metrics-   40 determine sharpness predictor step-   50 sharpness predictor-   100 low frequency content-   110 mixture of high horizontal and low vertical frequency content-   120 mixture of low horizontal and high vertical frequency content-   130 high frequency content

1. A digital camera incorporating a sharpness predictor, comprising: animage sensor; a user interface for displaying messages to a user; and adata processing system; and a memory system communicatively connected tothe data processing system and storing instructions configured to causethe data processing system to implement a method fordetermining-sharpness predictors for input digital images that are nottest pattern images, wherein the instructions comprise: capturing aninput digital image by photographing a natural scene using the imagesensor; dividing the input digital image into image blocks; using aprocessor to compute blocks of discrete cosine transform coefficients byapplying discrete cosine transforms to the image blocks; using aprocessor to compute one or more image metrics by analyzing only thesingle input digital image captured by the digital camera, wherein atleast one of the image metrics is an image sharpness metric related tothe spatial frequency content of the single input digital image which isdetermined by combining the discrete cosine transform coefficientscomputed for a plurality of image blocks; using a processor to compute asingle sharpness predictor for the single input digital image from theone or more image metrics, wherein the sharpness predictor represents anindication of the level of sharpness degradations introduced by thedigital camera; and displaying a message on the user interface when thecomputed sharpness predictor indicate that the input digital image lackssufficient sharpness.
 2. The digital camera of claim 1 wherein the inputdigital image is a color input digital image and one or more of theimage metrics are computed from a luminance signal computed from thecolor input digital image.
 3. The digital camera of claim 1 wherein thecomputation of the block metrics includes the step of computing imagestatistics for the blocks of discrete cosine transform coefficients. 4.The digital camera of claim 3 wherein the computation of the imagestatistics includes computing a centroid for the blocks of discretecosine transform coefficients.
 5. The digital camera of claim 3 whereinthe computation of the image statistics includes computing sums for theblocks of discrete cosine transform coefficients.
 6. The digital cameraof claim 3 wherein blocks of discrete cosine transform coefficientsfailing to meet a threshold criterion are excluded from the computationof the image metric.
 7. The digital camera of claim 1 wherein the stepof applying discrete cosine transforms to blocks of the input digitalimage is applied as part of an image compression operation.
 8. Thedigital camera of claim 1 wherein the image sharpness metric is computedfrom compressed digital image data determined using an image compressionoperation.
 9. The digital camera of claim 1 wherein the compresseddigital image data is stored in a digital image file.
 10. The digitalcamera of claim 1 wherein metadata providing an indication of the imagesharpness is stored in association with the digital image fileresponsive to the computed sharpness predictor.
 11. The digital cameraof claim 1 wherein the computation of one or more of the image metricsincludes the step of computing the number of image pixels darker than aspecified threshold.
 12. The digital camera of claim 1 wherein thecomputation of one or more of the image metrics includes the step ofcomputing the number of image pixels lighter than a specified threshold.13. The digital camera of claim 1 wherein the image metrics are computedfrom a subset of the input digital image.
 14. The digital camera ofclaim 13 wherein the subset of the input digital image is extracted fromthe center of the input digital image.
 15. The digital camera of claim13 wherein the subset of the input digital image corresponds to a mainsubject region of the input digital image.
 16. The digital camera ofclaim 15 wherein an automatic main subject detection algorithm is usedto identify the main subject region of the input digital image.
 17. Thedigital camera of claim 1 wherein the sharpness predictor is computed bycomputing a weighted summation of the two or more image metrics.
 18. Thedigital camera of claim 1 wherein the sharpness predictor is computed bycomputing a weighted sum of functions of the one or more image metrics.19. The digital camera of claim 1 further including using the sharpnesspredictor to estimate whether the quality of the input digital image issufficient for a particular usage.
 20. The digital camera of claim 19wherein the usage is producing a reflection print.
 21. The digitalcamera of claim 1 wherein the displayed message presents a recommendedimage usage to a user.
 22. The digital camera of claim 1 furtherincluding using the sharpness predictor as a control parameter for asubsequent image processing operation.
 23. The digital camera of claim 1wherein the step of computing the sharpness predictor further includesusing one or more additional sharpness metrics computed from imagecapture parameters.
 24. The digital camera of claim 23 wherein the imagecapture parameters include an exposure time parameter, a lens apertureparameter, a flash-fired parameter, or combinations thereof.
 25. Thedigital camera of claim 1 wherein the displayed message is a warningmessage which is displayed when the computed sharpness predictorindicates that unacceptable sharpness degradations have been introducedby the digital camera.